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[Document Name] CLAIMS 
[Claiml] 

An imaging apparatus comprising: 
a plurality of optical systems; 

an imaging element including a plurality of imaging regions that 
correspond to the plurality of optical systems in one-to-one correspondence; 

a pixel shift means for relatively shifting positions of images formed 
on the imaging element in time series; 

a memory for storing image information obtained by the imaging 
element; 

a shake amount obtaining means for obtaining a shake amount by 
comparing the image information stored in the memory; 

an optimal image selecting means for selecting images used for 
image combination from images stored in the memory; and 

an image combining means for combining a plurality of pieces of 
image information stored in the memory, 

wherein the pixel shift means does not perform a pixel shift on 
images formed by at least one optical system of the plurality of optical 
systems, but performs a pixel shift on images formed by the remaining 
optical systems, 

the imaging element captures images in each of the plurality of 
imaging regions with the same timing regardless of presence or absence of 
the pixel shift, 

the shake amount obtaining means obtains the shake amount of an 
image by comparing the plurality of pieces of image information captured in 
time series by the optical system without performing the pixel shift and 
stored in the memory, 
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the optimal image selecting means selects images used for image 
combination based on the shake amount obtained by the shake amount 
obtaining means, and 

the image combining means corrects the images selected by the 
optimal image selecting means based on the shake amount obtained by the 
shake amount obtaining means, and then combines the images. 
[Claim 2] 

The imaging apparatus according to claim 1, further comprising a 
subject discriminating means for discriminating different subjects, 

wherein the shake amount obtaining means obtains the shake 
amount of an image for each subject discriminated by the subject 
discriminating means, and 

the optimal image selecting means selects the images used for image 
combination for each subject discriminated by the subject discriminating 
means. 
[Claim 3] 

The imaging apparatus according to claim 1, further comprising a 
block dividing means for dividing an image into a plurality of blocks, 

wherein the shake amount obtaining means obtains the shake 
amount for each block divided by the block dividing means, and 

the optimal image selecting means selects the images used for image 
combination for each block divided by the block dividing means. 
[Claim 4] 

The imaging apparatus according to claim 1, further comprising a 
shift amount determining means for determining a shift amount for the 
pixel shift, 

wherein the shift amount determining means determines the shift 
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amount for the pixel shift for each image capture in time series based on the 
images selected by the optimal image selecting means and the shake 
amount obtained by the shake amount obtaining means, and 

the pixel shift means performs the pixel shift based on the shift 
amount determined by the shift amount determining means. 
[Claim 5] 

The imaging apparatus according to claim 1, further comprising a 
parallax obtaining means for obtaining a parallax amount between images 
captured by the plurality of optical systems, 

wherein the image combining means corrects the images selected by 
the optimal image selecting means based on the shake amount obtained by 
the shake amount obtaining means and the parallax amount obtained by 
the parallax obtaining means, and then combines the images. 
[Claim 6] 

The imaging apparatus according to claim 5, wherein the plurality of 
optical systems comprise an optical system for handling a red color, an 
optical system for handling a green color, and an optical system for handling 
a blue color, and at least one of the optical systems for handling the 
respective colors includes two or more optical systems, 

wherein the imaging apparatus further comprises a wavelength 
separating means that is located on an optical axis of the optical system for 
handling a red color and transmits the red color, a wavelength separating 
means that is located on an optical axis of the optical system for handling a 
green color and transmits the green color, and a wavelength separating 
means that is located on an optical axis of the optical system for handling a 
blue color and transmits the blue color, and 

wherein the pixel shift means does not perform a pixel shift on 
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images formed by at least one optical system of the two or more optical 
systems for handling the same color, but performs a pixel shift on images 
formed by the remaining optical systems. 
[Claim 7] 

The imaging apparatus according to claim 6, wherein each of the 
optical systems for handling the respective colors includes two or more 
optical systems, and 

the pixel shift means does not perform a pixel shift on images 
formed by at least one optical system of the two or more optical systems for 
handling the respective colors, but performs a pixel shift on images formed 
by the remaining optical systems. 
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[Document Name] SPECIFICATION 
[Title of the Invention] IMAGING APPARATUS 
[Technical Field] 
[0001] 

The present invention relates to an imaging apparatus having a 
pixel shift function. 
[Background Art] 

[0002] 

At present, a technique called "pixel shift" is known as a means for 
improving the resolution of an imaging element. FIG. 21 is a conceptual 
diagram for explaining how to improve the resolution using the pixel shift 
technique, and shows enlarged pixels of an imaging element. As illustrated 
in FIG. 21A, the imaging element includes an optical-to-electrical conversion 
portion 2101 (hereinafter referred to as an "optical-to-electrical conversion 
portion") which converts light into an electrical signal, and an invalid 
portion 2102 (hereinafter referred to as an "invalid portion"), such as a 
transfer electrode or the like, which cannot convert light into an electrical 
signal. The optical-to-electrical conversion portion and the invalid portion 
are combined into one pixel. In general, the pixels of the imaging element 
are regularly formed at predetermined intervals (pitches). A portion 
enclosed with a thick line in FIG. 21A is one pixel, and P indicates one pitch. 

[0003] 

In the pixel shift, first, an image is captured at a position of the 
imaging element illustrated in FIG. 21 A. Next, the imaging element is 
shifted, e.g., in a slanting direction by 1/2 pitch of a pixel in both the 
horizontal direction and the vertical direction so that the optical-to-electrical 
conversion portion is shifted to the invalid portion for each pixel, and then 
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another image is captured (FIG. 2 IB). Thereafter, these two captured 
images are electrically combined, taking into consideration the shift amount 
of the imaging element. 
[0004] 

In this manner, a signal can be taken out of the invalid portion from 
which no signal can be inherently derived. Thus, the imaged state of FIG. 
21C can be equivalent to that of an image captured by an imaging element 
with a double number of optical-to-electrical conversion portions, as 
compared to the imaged state of an image captured only once by the imaging 
element of FIG. 21A. Therefore, if the image is shifted as described above, 
an image equivalent to that captured by an imaging element having a 
double number of pixels can be obtained without increasing the number of 
pixels. 

[0005] 

As described above, the relative positional relationship between the 
light entering the imaging element and the imaging element is shifted, e.g., 
by 1/2 pixel, and images captured at the respective positions may be 
combined, resulting in a high-resolution image. 

[0006] 

The pixel shift is not limited to the slanting direction as described 
above, and the same effect can be obtained by the pixel shift in either the 
horizontal direction or the vertical direction. In addition to the shift by 1/2 
pixel, the same effect also can be obtained by shifting the pixels so as to 
interpolate their invalid portions. The resolution can be improved by 
increasing the number of times the images are captured at different places 
rather than by shifting the pixels only once. 

[0007] 
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Although, in the above-described example, the relative positional 
relationship between the imaging element and the incident light is changed 
by shifting the imaging element, the pixel shift is not limited thereto. For 
example, the optical lens may be moved instead of the imaging element. 
Alternatively, for example, another method has been proposed in which a 
parallel plate is employed (e.g., Patent Document l). In the invention of 
Patent Document 1, the parallel plate is tilted so as to shift the optical path 
of image light that forms an image on the imaging element, thereby 
performing the pixel shift. 

[0008] 

When the resolution is improved by the pixel shift, a plurality of 
images are captured in time series and then combined to produce a 
high-resolution image. In the pixel shift, it is necessary to eliminate a 
shake caused by the movement of a photographer such as a camera shake 
(hereinafter referred to as a "photographer shake") and a shake caused by 
the movement of a subject (hereinafter referred to as a "subject shake") 
during capturing the plurality of images in time series. 

[0009] 

If a shake occurs during capturing the images, and these images are 
combined without any modification, the resultant image may be 
significantly different from the original image. Therefore, some methods 
for removing a shake as much as possible or correcting a shake have been 
proposed. 

[0010] 

One method is to capture an image while fixing a camera using a 
tripod or the like. This method can reduce the influence of a photographer 
shake, but cannot reduce the influence of a subject shake. 
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[0011] 

Another method is to correct a shake with a shake detecting means. 
For example, the shake detecting means such as an angular velocity sensor 
is used to detect a photographer shake that occurs when the photographer 
presses a shutter release or the like, and then a shake amount is corrected 
based on the detected amount (e.g., Patent Document 2). 

[0012] 

In the invention of Patent Document 2, the pixel shift is performed 
by detecting a shake amount with the shake detecting means, correcting the 
pixel shift direction or the like based on the shake amount, and shifting the 
imaging element. This method can reduce the influence of a photographer 
shake. In addition to the method for shifting the imaging element, the 
same effect also can be obtained by a method for moving the optical lens or 
the like in accordance with the detected shake amount. However, even this 
method cannot detect a subject shake itself, and thus cannot reduce the 
influence of a subject shake. 

[0013] 

Yet another method is to determine a shake amount by comparing a 
plurality of images captured in time series and to correct a shake (e.g., 
Patent Document 3). In the invention of Patent Document 3, which was 
made by the present inventors, a shake amount is determined by comparing 
the plurality of images captured in time series, and a shake is corrected. 

[0014] 

To determine the shake amount accurately, the optical system that 
performs the pixel shift is separated from the optical system that does not 
perform the pixel shift. Then, an accurate shake amount is determined in 
the optical system that does not perform the pixel shift and used to correct 
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the shake amount in the optical system that performs the pixel shift. With 
this method, since both the photographer shake and the subject shake are 
determined as image shakes, the shake amount can be accurately 
determined without using the shake detecting means such as an angular 
velocity sensor separately. Thus, high resolution can be achieved by the 
pixel shift. 

[Patent Document l] JP H6 261236 A 

[Patent Document 2] JP H11225284 A 

[Patent Document 3] JP 2004-363868 A 
[Disclosure of the Invention] 
[Problems to be Solved by the Invention] 

[0015] 

As described above, a high-resolution image can be obtained by 
removing or correcting a shake in the pixel shift. In particular, the method 
as disclosed in Patent Document 3 is suitable for correcting the 
photographer shake and the subject shake simultaneously with high 
accuracy. However, the following problems arise when a shake is corrected 
after capturing the images. 

[0016] 

FIG. 22 is a diagram for explaining an example of a conventional 
pixel shift and shows enlarged pixels of an imaging element. In FIG. 22, 
reference numeral 2201 denotes an optical- to-electrical conversion portion 
and 2202 denotes an invalid portion. Hereinafter, consideration will be 
given to a pixel shift in which the pixels are shifted by an amount 
corresponding to 1/2 pixel pitch of the imaging element so as to interpolate 
the invalid portions. In FIG. 22, the shift amount of the pixel shift is 
expressed as a pixel shift vector 2203. Although the vector is indicated in 
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only the upper left pixel, all the pixels are shifted in the same manner. In 
FIG. 22, the shake amount determined during the image capture also is 
expressed as a shake vector. The method for determining the shake 
amount will be described in detail later. 
[0017] 

FIG. 22A shows a state in which no shake is present. Therefore, 
the pixel shift can be performed so as to interpolate the invalid portions of 
the pixels, as designed. Thus, a high-resolution image can be obtained by 
combining the images after the pixel shift. 

[0018] 

Even if a shake vector 2204 is present as shown in FIG. 22B, since 
the sum of the pixel shift vector 2203 and the shake vector 2204 is located in 
the invalid portion, a shake correction can be performed to provide pixel 
shift information so as to interpolate the invalid portions of the pixels. 
Thus, a high-resolution image can be obtained by combining the images 
after the pixel shift. 

[0019] 

However, when a shake vector 2205 is present as shown in FIG. 22C, 
the sum of the pixel shift vector 2203 and the shake vector 2205 is located in 
the optical-to-electrical conversion portion (or in the vicinity of the 
optical- to -electrical conversion portion) of the imaging element. Therefore, 
even if the shake correction is precisely performed, only the information 
that has been already available can be provided. Thus, a high-resolution 
image cannot be achieved even by the image combination. 

[0020] 

FIG. 23 shows an optical system that does not perform a pixel shift, 
and a shake is present in the optical system as in the case of FIG. 22. In 
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FIG. 23A, no shake is present. In FIG. 23B, a shake is present in the same 
amount as the vector 2204 in FIG. 22B. In FIG. 23C, a shake is present in 
the same amount as the vector 2205 in FIG. 22C. 
[0021] 

As is evident from FIG. 23C, in the optical system that does not 
perform the pixel shift, the invalid portions of the original pixels are 
interpolated by the optical-to-electrical conversion portions of the imaging 
element, even if the shake is present to the extent that the pixel shift is 
hardly effective as shown in FIG. 22C. 

[0022] 

Whether the shake is present as represented by the vector 2204 or 
the vector 2205 depends on the probability and cannot be intentionally 
controlled by a photographer. Therefore, when the captured images are 
combined after the shake correction in expectation of a high-resolution 
image due to the pixel shift, there is a possibility that the resultant image 
does not actually have such high resolution. 

[0023] 

The present invention is intended to solve the above conventional 
problems and has an object of providing an imaging apparatus that 
performs a pixel shift and can prevent a reduction in the effect of the pixel 
shift even if there is a photographer shake or a subject shake. 
[Means for Solving the Problems] 

[0024] 

To achieve the above object, the imaging apparatus of the present 
invention includes the following" a plurality of optical systems; an imaging 
element including a plurality of imaging regions that correspond to the 
plurality of optical systems in one-to-one correspondence! a pixel shift 
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means for relatively shifting positions of images formed on the imaging 
element in time series! a memory for storing image information obtained by 
the imaging element; a shake amount obtaining means for obtaining a 
shake amount by comparing the image information stored in the memory; 
an optimal image selecting means for selecting images used for image 
combination from images stored in the memory; and an image combining 
means for combining a plurality of pieces of image information stored in the 
memory. The pixel shift means does not perform a pixel shift on images 
formed by at least one optical system of the plurality of optical systems, but 
performs a pixel shift on images formed by the remaining optical systems. 
The imaging element captures images in each of the plurality of imaging 
regions with the same timing regardless of presence or absence of the pixel 
shift. The shake amount obtaining means obtains the shake amount of an 
image by comparing the plurality of pieces of image information captured in 
time series by the optical system without performing the pixel shift and 
stored in the memory. The optimal image selecting means selects images 
used for image combination based on the shake amount obtained by the 
shake amount obtaining means. The image combining means corrects the 
images selected by the optimal image selecting means based on the shake 
amount obtained by the shake amount obtaining means, and then combines 
the images. 

[Effect of the Invention] 
[0025] 

The present invention can prevent a reduction in the effect of the 
pixel shift even if a photographer shake or a subject shake is present during 
the pixel shift, and thus achieve a high-resolution image. 
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[Best Modes for Carrying Out the Invention] 
[0026] 

The present invention can improve the reliability of acquiring 
information for interpolating the invalid portions of the pixels of the 
imaging element, even if the photographer shake or the subject shake is 
present during the pixel shift. Therefore, the present invention can 
prevent a reduction in the effect of the pixel shift and achieve a 
high-resolution image. 

[0027] 

The imaging apparatus may further include a subject discriminating 
means for discriminating different subjects. It is preferable that the shake 
amount obtaining means obtains the shake amount of an image for each 
subject discriminated by the subject discriminating means, and the optimal 
image selecting means selects the images used for image combination for 
each subject discriminated by the subject discriminating means. 

[0028] 

With this configuration, even if the shake amount of the whole 
image and the shake amount of the subject are different, the shake amount 
can be precisely obtained, and thus the optimal images for image 
combination can be properly selected. 

[0029] 

The imaging apparatus may further include a block dividing means 
for dividing an image into a plurality of blocks. It is preferable that the 
shake amount obtaining means obtains the shake amount for each block 
divided by the block dividing means, and the optimal image selecting means 
selects the images used for image combination for each block divided by the 
block dividing means. 
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[0030] 

With this configuration, the shake amount is obtained for each block, 
and the optimal image is also selected for each block. Therefore, the whole 
image can have high resolution. Moreover, if the size of the blocks is set as 
desired, a photographer can choose that the image combination time is 
reduced, or the resolution of the image is more improved in accordance with 
his/her intention. 
[0031] 

The imaging apparatus may further include a shift amount 
determining means for determining a shift amount for the pixel shift. It is 
preferable that the shift amount determining means determines the shift 
amount for the pixel shift for each image capture in time series based on the 
images selected by the optimal image selecting means and the shake 
amount obtained by the shake amount obtaining means, and the pixel shift 
means performs the pixel shift based on the shift amount determined by the 
shift amount determining means. 

[0032] 

With this configuration, the image to be selected by the optimal 
image selecting means can be reliably obtained, so that high resolution also 
can be ensured. 

[0033] 

The imaging apparatus may further include a parallax obtaining 
means for obtaining a parallax amount between images captured by the 
plurality of optical systems. It is preferable that the image combining 
means corrects the images selected by the optimal image selecting means 
based on the shake amount obtained by the shake amount obtaining means 
and the parallax amount obtained by the parallax obtaining means, and 
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then combines the images. 
[0034] 

With this configuration, even if a plurality of images captured by 
different optical systems and selected by the optimal image selecting means 
are to be combined, the image combination can be performed after correcting 
both the shake amount and the parallax amount. Thus, a high-resolution 
image can be obtained. 

[0035] 

It is preferable that the plurality of optical systems includes an 
optical system for handling a red color, an optical system for handling a 
green color, and an optical system for handling a blue color, and at least one 
of the optical systems for handling the respective colors includes two or 
more optical systems. The imaging apparatus may further include a 
wavelength separating means that is located on an optical axis of the optical 
system for handling a red color and transmits the red color, a wavelength 
separating means that is located on an optical axis of the optical system for 
handling a green color and transmits the green color, and a wavelength 
separating means that is located on an optical axis of the optical system for 
handling a blue color and transmits the blue color. It is also preferable that 
the pixel shift means does not perform a pixel shift on images formed by at 
least one optical system of the two or more optical systems for handling the 
same color, but performs a pixel shift on images formed by the remaining 
optical systems. 

[0036] 

This configuration can produce a high -re solution full color image in 
which both shake and parallax have been corrected. 
[0037] 
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In this case, it is preferable that each of the optical systems for 
handling the respective colors includes two or more optical systems, and the 
pixel shift means does not perform a pixel shift on images formed by at least 
one optical system of the two or more optical systems for handling the 
respective colors, but performs a pixel shift on images formed by the 
remaining optical systems. This configuration can ensure the effect of the 
pixel shift and becomes more advantageous to high resolution. 

[0038] 

Hereinafter, an embodiment of the present invention will be 
described with reference to the accompanying drawings. 
[0039] 

(Embodiment l) 

FIG. 1 is a block diagram illustrating a configuration of an imaging 
apparatus according to Embodiment 1. A system control means 101 is a 
central processing unit (CPU) that controls the whole imaging apparatus. 
The system control means 101 controls a pixel shift means 102, a transfer 
means 103, a memory 107, a shake amount obtaining means 108, an 
optimal image selecting means 109, and an image combining means 110. 

[0040] 

A subject (not shown) is imaged in different regions of an imaging 
element 106 by imaging optical systems 104, 105, and then photoelectrically 
converted into image information as a light intensity distribution by the 
imaging element 106. The pixel shift means 102 shifts the relative position 
between the imaging element 106 and the subject image formed on the 
imaging element 106 by the imaging optical system 105. The relative 
position between the imaging optical system 104 and the imaging element 
106 is designed not to be changed. The transfer means 103 transfers the 
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image information provided by the imaging element to the memory 107 that 
stores images. 
[0041] 

The images are captured at different times (in time series) while the 
pixel shift is performed, and image information captured using the imaging 
optical system 104 and pixel-shifted image information captured using the 
imaging optical system 105 are successively stored in the memory 107 every 
time the images are captured. 

[0042] 

The shake amount obtaining means 108 obtains a shake amount by 
comparing the image information captured at different times (in time series) 
using the imaging optical system 104, i.e., the optical system that does not 
perform the pixel shift. The optimal image selecting means 109 selects the 
optimal images for image combination based on the shake amount obtained 
by the shake amount obtaining means 108. 

[0043] 

The image combining means 110 combines the images selected by 
the optimal image selecting means 109 after the correction based on the 
shake amount obtained by the shake amount obtaining means 108, thereby 
producing a high-resolution image. 

[0044] 

FIG. 2 is a flowchart illustrating a whole operation of the imaging 
apparatus of this embodiment. The image capture is started by an image 
capture start command of step 200. When the image capture is started, an 
image capture preprocess of step 201 initially is performed. In this process, 
an optimal exposure time is calculated, and a focusing process is performed. 

[0045] 
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For example, there is a phenomenon that if a distance between a 
subject and an imaging apparatus is changed, an imaging distance is 
changed, so that an image is blurred. To correct the phenomenon, a 
distance between an imaging optical system and an imaging element is 
adjusted (focusing). Focusing can be achieved by using a property that a 
captured image has a maximum contrast when a focus is obtained, and 
changing a space (imaging distance) between the imaging optical system 
and the imaging element using a focusing actuator (not shown). 

[0046] 

Note that contrast does not have to be used necessarily for focusing, 
and a distance to a subject may be measured using laser, radio wave, or the 
like, to perform focusing. 

[0047] 

Also, an optimal exposure time needs to be adjusted, taking ambient 
light into consideration. To this end, there are a method of detecting 
brightness using an illuminance sensor to set an exposure time, a method of 
providing a preview function that takes in an image before the start of the 
image capture, and the like. In the case of the method of providing the 
preview function, an image taken in before the start of the image capture is 
converted into a grayscale image (i.e., brightness information). Thereafter, 
if the histogram is unbalanced to a white color (bright), it is determined that 
exposure is excessive (excessively long exposure time), and if the histogram 
is unbalanced to a black color (dark), it is determined that exposure is 
insufficient (excessively short exposure time), and the exposure time is 
adjusted. 

[0048] 

When the preview function is possessed, by performing this 
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preprocess before an image capture start command, a time required from 
the image capture start command to the start of exposure can be reduced. 
[0049] 

Next, the image capture is performed in step 202. The image 
capture is performed by repeating the processes of steps 203 to 205. In step 
203, the imaging element 106 performs a photoelectric conversion (i.e., an 
exposure process). In step 204, the transfer means 103 transfers an image 
from the imaging element 106 to the memory 107. In step 205, the pixel 
shift means 102 shifts the relative position between the imaging optical 
system 105 and the imaging element 106. 

[0050] 

As described above, when the images are captured at different times, 
a shake may occur in the images due to the photographer shake or the 
subject shake during each image capture. In step 206, to correct the shake, 
the shake amount obtaining means 108 determines a shake amount from a 
plurality of pieces of image information that have been captured in time 
series by the imaging optical system 104 and stored in the memory. 
[0051] 

The presence of the photographer shake and the subject shake 
causes the images to move. Therefore, when one of two images captured at 
different times is identified as a comparative reference image and the other 
is identified as a comparative target image, the amount of photographer 
shake and the amount of subject shake can be determined by examining to 
which part of the comparative target image a certain region of the 
comparative reference image is moved. 

[0052] 

Hereinafter, a method for determining the shake amount by image 
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comparison will be described. In order to examine to which region of the 
comparative target image a specific region of the comparative reference 
image (hereinafter referred to as a "comparative reference region") 
corresponds, an evaluation region having the same size as the comparative 
reference region is set in the comparative target image to evaluate how 
much the comparative reference region and the evaluation region resemble 
each other. Thereafter, evaluation regions are set successively at other 
positions, and the destination of the comparative reference region is 
searched for while performing the evaluation in each of the evaluation 
regions. In this case, the evaluation region that resembles the comparative 
reference region most is the destination of the comparative reference region. 
[0053] 

An image captured by the imaging element can be considered as a 
set of light intensities corresponding to the individual pixels. Therefore, if 
an upper left of the image is the origin, and the light intensity of a pixel 
which is an x-th pixel to the right in the horizontal direction and a yth pixel 
downward in the vertical direction is represented by I(x, y), the image can 
be assumed to be a distribution of the light intensity I(x, y). 

[0054] 

FIG. 3 illustrates the positional relationship between a comparative 
reference region 301 and an evaluation region 302. In the example of FIG. 
3, the comparative reference region is set to be in the shape of a rectangle 
which has an upper left pixel located at (xl, yl) and a lower right pixel 
located at (x2, y2). In this case, an evaluation region (m, n) shifted by m 
pixels in the right direction and n pixels in the downward direction from the 
comparative reference region can be represented by a region having an 
upper left pixel located at (xl+m, yl+n) and a lower right pixel located at 
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(x2+m, y2+n). 
[0055] 

An evaluation value R(m, n) indicating a correlation between the 
evaluation region and the comparative reference region (how much they 
resemble each other) is represented by the sum of the absolute values of 
differences in light intensity between each pixel, as indicated by (Expression 
1). 

[0056] 

[Expression l] 

R{m,ri)= £ ^\l l (x 9 y)-I 2 (x + m 9 y + n)\ 

y=y\ x = x \ 

[0057] 

The more the comparative reference region and the evaluation 
region resemble each other, the smaller the difference in light intensity 
between corresponding pixels in the two regions. Therefore, the evaluation 
value R(m, n) indicates a smaller value as the correlation between the light 
intensity distributions (image) of the comparative reference region and the 
evaluation region increases (they resembles each other to higher extent). 

[0058] 

In the present invention, the shake amount should be determined 
with an accuracy of not more than a pixel pitch of the imaging element 106 
(referred to as "subpixel" in the following) so as to select the images by the 
optimal image selecting means 109. As a method for determining the 
shake amount with subpixel accuracy, data F(x, y) is newly created by 
interpolating between each pixel from the original light intensity I(x, y), and 
based on F(x, y), the evaluation value R(m, n) is calculated in accordance 
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with (Expression l). The interpolation of data may be either linear 
interpolation or nonlinear interpolation. 
[0059] 

Thus, the shake amount is obtained by searching for the evaluation 
region that resembles the comparative reference region most with subpixel 
accuracy while changing the values of m and n. In this case, since the 
shake directions of the photographer shake and the subject shake are not 
limited to specific directions, negative values also need to be studied for the 
values of m and n (evaluation of regions shifted in the left direction or in the 
upward direction). 

[0060] 

m and n may be changed so that the whole range of the comparative 
target image can be evaluated. However, when the image of a subject is 
moved significantly due to an apparatus shake, so that the image is 
departed from the light receiving range of the imaging element, the subject 
cannot be combined as an image. Therefore, in general, it is preferable 
that m and n be limited to a predetermined range, thereby reducing a 
calculation time. A combination of m and n thus found that minimizes the 
evaluation value R(m, n) is a shake amount indicating the position of a 
region of a comparative target image corresponding to the comparative 
reference region. 

[0061] 

Note that the comparative reference region is not necessarily limited 
to rectangles, and can be set to be in any arbitrary shape. Also, the 
calculation of the evaluation value is not necessarily limited to the sum of 
the absolute values of differences in light intensity. Any function 
indicating the correlation (e.g., normalization is performed for each region 
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before obtaining the correlation, etc.) may be used to calculate the 
evaluation value. 
[0062] 

Next, in step 207, the optimal image selecting means 109 selects the 
optimal images for image combination based on the shake amount obtained 
by the shake amount obtaining means 108. The method for selecting the 
optimal images can be determined by user's setting. As one possible 
method, the shake amount obtained by the shake amount obtaining means 
108 may be expressed as a shake vector 401, as shown in FIG. 4A. In this 
case, the shake vector 401 is divided into a vector 402 and a vector 403. 
The vector 402 extends to the optical-to-electrical conversion portion that is 
closest to the end point of the shake vector 401 and the vector 403 extends 
from the closest optical-to-electrical conversion portion to the end point of 
the shake vector 401. 

[0063] 

FIG. 4B is an enlarged view showing a pixel including the vector 403 
and other pixels around this pixel. Reference numeral 404 denotes a region 
of 1/4 pixel pitch that has the base point of a pixel pitch as a center and is 
aligned in both vertical and horizontal directions. Reference numeral 405 
denotes a region of 1/2 pixel pitch that has the base point of a pixel pitch as 
a center and is aligned in both vertical and horizontal directions. 

[0064] 

In this embodiment, when the absolute value of the vector 403 is in 
the range of 0 to 1/4 pixel pitch, the images captured by the optical system 
105 that performs the pixel shift are selected. When the shake amount is 
in the range of 1/4 to 1/2 pixel pitch, the images captured by the optical 
system 104 that does not perform the pixel shift are selected. This ensures 
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the image information for interpolating the invalid portions of the images 
that are captured before a shake occurs. On the other hand, if a single 
imaging optical system is used, whether or not such image information for 
interpolating the invalid portions can be obtained depends on the probability, 
and thus the reliability becomes poor. 
[0065] 

In the example of FIG. 4, since the shake vector 403 is in the range 
of 1/4 to 1/2 pixel pitch, the optimal image selecting means 109 selects the 
images of the optical system 104 that does not perform the pixel shift. 

[0066] 

In this embodiment, although the optical images are selected based 
on the pixel pitch in the vertical and horizontal directions of the imaging 
element, the pixel pitch in the slanting direction also may be used. 
Moreover, some references of the pixel pitch may be used together according 
to the circumstances. 

[0067] 

Next, in step 208, the images that have been stored in the memory 
107 and selected by the optimal image selecting means 109 are corrected 
based on the shake amount determined in step 206 and then combined, 
thereby producing a high-resolution image. 

[0068] 

FIG. 5 is a conceptual diagram of a method for correcting the shake 
amount. Images 501 to 504 are stored in the memory and captured at two 
different times (referred to as "image capture time 1" and "image capture 
time 2" in the following) by the imaging optical system 104 and the imaging 
optical system 105. 

[0069] 
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The image 501 is captured by the imaging optical system 104 at the 
image capture time 1. The image 502 is captured by the imaging optical 
system 105 at the image capture time 1. The image 503 is captured by the 
imaging optical system 104 at the image capture time 2. The image 504 is 
captured by the imaging optical system 105 at the image capture time 2. A 
region 501a is within the image 501. In the image 502, a region 502a 
contains the same image as the subject image present in the region 501a. 

[0070] 

In the image 503 captured at the image capture time 2, the subject 
image present in the region 501a is moved to a region 503c. A vector 503b 
is a difference between the region 503c and a region 503a that is located at 
the original position of the region 503c (i.e., the same position as the region 
501a), and indicates the shake amount of the region 501a due to the 
photographer shake or the subject shake. 

[0071] 

Similarly, in the image 504 captured at the image capture time 2, 
the subject image present in the region 502a is moved to a region 504e. 
Since the pixel shift is performed on the image 504, a vector 504f is a 
difference between the region 504e and a region 504a that is located at the 
original position of the region 504e (i.e., the same position as the region 
502a). Moreover, the vector 504f is the sum of a vector 504b that indicates 
the shake amount due to the photographer shake or the subject shake and a 
vector 504d that is intentionally shifted by the pixel shift. 

[0072] 

In the case of a normal pixel shift, the image 504e is to be combined. 
Therefore, the image (504c) that is initially supposed to be extracted cannot 
be precisely extracted. Thus, the image is degraded. 
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[0073] 

However, the timing of the image capture is the same in both the 
imaging optical system 104 and the imaging optical system 105. Therefore, 
there is no difference in the shake amount due to the photographer shake or 
the subject shake between the image 503 and the image 504 (i.e., the vector 
503b and the vector 504b are equal). Accordingly, when the shake amount 
(vector 503b) derived from the images 501, 503 is taken into account in 
performing the pixel shift and the image combination, the images can be 
properly combined. Thus, a high-resolution image can be obtained. 

[0074] 

As described above, the optical system that does not perform the 
pixel shift is allowed to capture an image at the same time as the image 
capture by the optical system while it performs the pixel shift. Using this 
image as a reference, a shake such as the photographer shake or the subject 
shake can be precisely corrected. 

[0075] 

The images that have been selected by the optimal image selecting 
means 109 and stored in the memory 107 are corrected with the above 
correction method, and then combined by the image combining means 110, 
so that a high-resolution image can be produced without any degradation of 
the image. 

Finally, in step 209, the combined image is output, and a series of 
image capture operations is ended. Hereinafter, Embodiment 1 will be 
described more specifically by way of the following examples. 

[0076] 

(Example l) 

FIG. 6 shows a configuration including the imaging optical system, 
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the pixel shift means, and the imaging element of this example. As the 
imaging optical system, two aspherical lenses 601a and 601b, each having a 
diameter of 3 mm, were used. The optical axes of the lenses are 
substantially parallel to the Z-axis in FIG. 6. 
[0077] 

A glass plate 602 was located on the optical axis of the lens 601b and 
tilted by a piezoelectric actuator and a tilting mechanism. With this 
configuration, the pixel shift was performed. The piezoelectric actuator 
and the tilting mechanism are not shown in FIG. 6. As the glass plate 602, 
an optical glass BK7 having a width (X-axis direction in FIG. 6) of 1 cm, a 
height (Y-axis direction in FIG. 6) of 1 cm, and a thickness (Z-axis direction 
in FIG. 6) of 500 \im was used. 

[0078] 

As an imaging element 603, a black-and-white CCD 603 was used in 
which a pitch between adjacent pixels was 2.4 jam. The glass plate 602 and 
the imaging element 603 had light receiving surfaces that were 
substantially parallel to the XY plane of FIG. 6. Also, the imaging element 
603 was divided into two regions 603a and 603b, which corresponded to the 
respective optical systems in one-to-one correspondence. To perform the 
pixel shift by 1/2 (1.2 |xm) of a pixel pitch, the glass plate should be tilted at 
about 0.4 degrees. In this configuration, an image obtained after one-time 
pixel shift, which allowed the pixels to be shifted to the invalid potions of 
the imaging element, and the original image were combined into an image. 

[0079] 

FIG. 7 shows images stored in the memory with this configuration. 
The time when a first image was captured is referred to as image capture 
time 1 and the time when a second image was captured after the pixel shift 
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(after the glass plate was tilted) is referred to as image capture time 2. 
[0080] 

Images 701 and 702 were captured through the lenses 601a and 
601b at the image capture time 1, respectively. An image 703 was captured 
through the lens 601a (i.e., the optical system that did not perform the pixel 
shift) at the image capture time 2. An image 704 was captured through the 
lens 601b (i.e., the optical system that performed the pixel shift) at the 
image capture time 2. 

[0081] 

In this example, a scene in which the motion of a subject is 
sufficiently small (e.g., a landscape, etc.) was captured. Therefore, there 
was no subject shake between the image 701 captured at the image capture 
time 1 and the image 703 captured at the image capture time 2. If a shake 
was present, it could be assumed that the whole image was moved due to 
the photographer shake. 

[0082] 

In this case, since the whole image was uniformly moved under 
certain conditions, the shake amount could be determined only by 
evaluating to which region of the image 703 (captured at the image capture 
time 2 by the optical system that did not perform the pixel shift) the central 
portion (e.g., a region of 100 x 100 pixels) of the image 701 (captured at the 
image capture time 1 by the same optical system) was moved, using an 
image comparing method with (Expression l). The size of the region to be 
compared may be set as desired. 

[0083] 

Based on this shake amount, the images were selected by the 
optimal image selecting means 109 for image combination. In this example, 
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the optimal images were selected by the method as described with reference 
to FIG. 4. The absolute value of the shake amount corresponding to the 
vector 403 in FIG. 4 was 0.1 pixel pitch. Therefore, the images captured by 
the optical system that performed the pixel shift were selected in this 
example. 

[0084] 

Next, the image 704 captured by the optical system that performed 
the pixel shift was corrected based on the above shake amount, and then 
combined with the image 702, resulting in a high-resolution image. 

[0085] 

In this example, although a method for tilting the glass plate was 
used as the pixel shift means, the pixel shift means is not limited to this 
method. For example, an actuator including a piezoelectric element, an 
electromagnetic actuator, or the like may be used to physically move the 
imaging element or the lenses in a predetermined amount. 

[0086] 

In this example, the glass plate was not provided for the optical 
system that did not perform the pixel shift. However, if the glass plate is 
provided so as not to be tilted, the pixel shift will not be performed. Such a 
configuration can increase the degree of freedom in deciding whether or not 
to perform the pixel shift, as compared to the case where the pixel shift 
means is excluded from the beginning. 

[0087] 

The same configuration may be applied to the pixel shift using any 
methods other than the glass plate. For example, if the pixel shift means is 
an actuator including a piezoelectric element, the pixel shift can be avoided 
by intentionally keeping the actuator from moving, although the actuator 
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may be excluded from the beginning so that the pixel shift is not performed. 
[0088] 

Like the parallel plate as described above, this case also increases 
the degree of freedom in deciding whether or not to perform the pixel shift, 
compared to the case where the pixel shift means is excluded from the 
beginning. 

[0089] 

In this example, although one imaging element was divided into two 
regions, two different imaging elements may be used for the respective 
optical systems in one-to-one correspondence. Any form of the imaging 
element may be used as long as a plurality of imaging regions correspond to 
the respective optical systems in one-to-one correspondence. 

[0090] 

FIG. 8 is a conceptual diagram of the photographer shake. FIG. 8A 
is a diagram illustrating the case where a subject and a camera are 
translated, and A of FIG. 8C shows a change between images at the image 
capture times 1 and 2. FIG. 8B is a diagram illustrating the case where 
the camera is horizontally rotated, and B of FIG. 8C shows a change 
between images at the image capture times 1 and 2. 

[0091] 

As can be seen from FIG. 8C, the influence of the photographer 
shake on the images is larger when the optical axis deviates due to the 
horizontal rotation of the camera, as shown in FIG. 8B, than when the 
subject and the camera are translated, as shown in FIG. 8A. FIG. 8B 
shows an example in which the camera is horizontally rotated, and the same 
is true for the camera to be vertically rotated. 

[0092] 



26 



In the case of rotation, since the distance between the subject and 
the lens is partially changed, a slight distortion occurs in the image. If the 
slightly distorted images are simply superimposed, portions that should 
coincide with each other do not coincide with each other, so that the effect of 
improving the resolution cannot be obtained even by the pixel shift. 

[0093] 

Therefore, the image distortion caused by the rotation needs to be 
detected and corrected. In the method of Example 1, the amount of 
photographer shake is determined in one specific evaluation region. 
However, if a plurality of evaluation regions are set, and the shake amount 
is determined in each of the evaluation regions, the amount of photographer 
shake and the image distortion can be obtained in each of the evaluation 
regions. At least two evaluation regions are required to detect the image 
distortion. By deforming the images to be superimposed in accordance 
with the image distortion, it is possible to prevent the degradation of the 
images and to produce a high-resolution image. 

[0094] 

(Example 2) 

Example 2 is different from Example 1 in that there is a shift 
amount of a subject whose image is to be captured (e.g., a human, an animal, 
etc.). In Example 2, a scene is captured in which a subject moves to 
another place during a time from capturing of a first image until the data is 
stored into a memory and a second image is captured, so that a portion of 
the subject moves to another place between the first image and the second 
image. 

[0095] 

The basic configuration of Example 2 is similar to that of Example 1, 
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and the overlapping portions will not be described. When a subject moves 
significantly, the whole image is not uniformly moved under certain 
conditions. Therefore, unlike Example 1, the whole movement cannot be 
estimated from the movement of some regions of the image. 
[0096] 

Therefore, in Example 2, a block dividing means is provided for 
dividing an image into a plurality of blocks, and a shake amount is 
determined for each block. The block dividing means is controlled by the 
system control means 101 and divides the whole of a first image captured by 
the optical system that does not perform the pixel shift into blocks having 
10x10 pixels. The shake amount obtaining means 108 examines to which 
position of a second image each of the blocks of the first image corresponds. 
The shift amount of the image was obtained by (Expression l). 

[0097] 

FIG. 9 shows the images stored in the memory in Example 2. FIG. 
9A illustrates an image captured at the image capture time 1. FIG. 9B 
illustrates an image captured at the image capture time 2 by the optical 
system that did not perform the pixel shift. FIG. 9C illustrates a shift 
amount of an image obtained for each block. 

[0098] 

In FIG. 9C, A represents a block for which a shake of 10.1 pixels in 
the right direction was obtained in FIG. 9A, and B represents a block in 
which a shake of 8.8 pixels in the left direction was obtained in FIG. 9A. 
FIG. 9 does not include a drawing that corresponds a detailed shake 
amount. 

[0099] 

Next, the optimal image selecting means 109 selected the images for 
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image combination. The images were selected in the same manner as 
Example 1. For the blocks represented by A in FIG. 9, the absolute value of 
the amount corresponding to the vector 403 in FIG. 4 was 0.1 pixel pitch. 
Therefore, the images captured by the optical system that performed the 
pixel shift were selected. 
[0100] 

For the blocks represented by B in FIG. 9C, the absolute value of the 
amount corresponding to the vector 403 in FIG. 4 was 0.2 pixel pitch. 
Therefore, the images captured by the optical system that performed the 
pixel shift were selected. 

[0101] 

As described above, when the image is divided into blocks, and the 
movement is recognized for each block, distributions of the shake amounts 
due to the photographer shake and the subject shake can be separately 
obtained. Moreover, the optimal images for image combination can be 
selected based on the shake amount in each block. 

[0102] 

Moreover, by setting the size of the blocks divided by the block 
dividing means as desired, the optical image combination can be achieved in 
accordance with the intention of a photographer. For example, if the image 
is divided into large blocks, the time required for image combination can be 
reduced. On the contrary, if the image is divided into small blocks, the 
shake amount can be determined in detail for each block, and thus the 
combined image has higher resolution. 

[0103] 

Next, the images selected by the optimal image selecting means 
were corrected based on the shake amount in each block obtained by the 
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shake amount obtaining means 108 and then combined, so that a 
high-resolution image was produced even if the subject moved significantly. 
[0104] 

Note that image processing may be performed in accordance with 
the selection by the user so that only the photographer shake is corrected 
and the subject shake is intentionally not corrected, thereby making it 
possible to provide a correction mode in which the dynamism of a scene 
having a motion is emphasized. 

[0105] 

In Example 2, all the images selected by the optimal image selecting 
means were captured by the same optical system. Therefore, the images 
captured by the same optical system were combined after only the correction 
based on the shake amount. However, depending on the absolute value of 
the amount corresponding to the vector 403 in FIG. 4, the images captured 
by the other optical system that performed the pixel shift could be selected. 
In other words, the images selected by the optimal image selecting means 
109 may differ in the optical system from block to block. 

[0106] 

A parallax may occur between the images captured by different 
optical systems, as will be described in Embodiment 2. The parallax can be 
corrected by a configuration of Embodiment 2, which will be described later. 
Thus, even if the images captured by different optical systems are selected, 
those images can be combined after correcting the parallax, thereby 
producing a high-resolution image. 

[0107] 

If the subject shake is present, there may be an occluded portion of 
the subject in the images after the pixel shift (i.e., the block indicated by "x" 
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in FIG. 9C). In such a case, only the image captured at a specific time may 
be selected without combining the images after the pixel shift for that 
portion. 

[0108] 

If a user selects blocks that will probably move and blocks that will 
not probably move before the image capture, the calculation cost of the 
shake amount after the image capture can be reduced. 

[0109] 

(Example 3) 

Example 3 is different from Examples 1, 2 in that a subject 
discriminating means for discriminating different subjects in an image is 
provided. By using the subject discriminating means, the shake amount 
can be easily obtained for each subject. Therefore, even when there are 
different shake amounts in an image, e.g., when not only the photographer 
shake but also the subject shake is present, the shake amounts can be 
precisely obtained. 

[0110] 

As described in Example 2, when the shake amount is determined 
by dividing the image into blocks, the division of the image into blocks can 
be performed for each subject, or the size of the blocks can be changed for 
each subject. For the image combination, the images may be combined 
selectively only for a specific subject. 

[0111] 

Examples of the subject discriminating means include a means for 
measuring a distance to a subject using radio wave to identify different 
image regions and a means for performing edge detection or the like using 
image processing to discriminate different subjects. The subject 
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discriminating means is not limited thereto. Any specific means can be 
used as long as different subjects in an image can be discriminated. The 
basic configuration of Example 3 is similar to that of Example 1, and 
overlapping portions will not be described. 
[0112] 

FIG. 10 is a diagram illustrating a captured image and subject 
groups discriminated by the subject discriminating means in Example 3. 
In Example 3, the captured image was divided into blocks having 10x10 
pixels (11 (width) x 9 (length)), and a distance to a subject was measured 
using radio wave for each block to discriminate different subjects. In the 
subject discrimination, subjects having a measured distance within a 
certain error range were discriminated as the same subject. In Example 3, 
the error range was 5%. 

[0113] 

FIG. 10A illustrates an image captured at the image capture time 1. 
FIG. 10B illustrates an image captured at the image capture time 2 without 
performing the pixel shift. Also, a distance (unit: meter) measured using 
radio wave is indicated for each block. 

[0114] 

Before the image capture at the image capture time 1, a distance to 
a subject was measured using radio wave. As a result, as illustrated in 
FIG. 10A, two subject groups were discriminated roughly. One is a subject 
group 1 at a distance of about 5 meters, and the other is a subject group 2 at 
a distance of about 2 meters. Each subject group is discriminated as falling 
within the above-described error range of 5%. 

[0115] 

Before the image capture at the image capture time 2, a distance to 
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a subject was measured using radio wave. As a result, subject groups were 
discriminated as illustrated in FIG. 10B. In Example 3, the shake amounts 
before and after pixel shift were obtained for each subject group. 
[0116] 

By using the shake amount obtaining means 108, the shake amount 
was obtained for each subject group. As a result, regarding the subject 
group 1, a shake of 10.3 pixel pitch in the left direction was obtained in FIG. 
10. This shake is illustrated as a 1-block shake in FIG. 10. Regarding the 
subject group 2, there is a large subject shake, and a portion of the subject 
group 2 is out of the image, so that the shake amount of the whole subject 
group was not precisely obtained. 

[0117] 

Therefore, in Example 3, in the image captured at the image capture 
time 2, the shake correction was performed only on the subject group 1 
before image combination. The images were selected by the optimal image 
selecting means 109 in the same manner as Example 1. 

[0118] 

The absolute value of the amount corresponding to the vector 403 in 
FIG. 4 was 0.3 pixel pitch. Therefore, the optimal image selecting means 
109 selected the images captured by the optical system that did not perform 
the pixel shift. In the subject group 1, the images selected by the optimal 
image selecting means 109 were combined. In the other portions, images 
were produced using the images captured at the image capture time 1 by 
the optical system that did not perform the pixel shift. 

[0119] 

As described in Example 3, by discriminating different subjects 
using the subject discriminating means, the shake amount can be obtained 
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for each subject, and thus the shake amount of an image can be precisely 
corrected. 

[0120] 

When a portion of an image is out of the image capture range due to 
the photographer shake and the subject shake so that the image cannot be 
recognized, the improvement of the resolution due to pixel shift is not 
performed in the image region, and one of a plurality of captured images 
may be selected. 

[0121] 

(Embodiment 2) 

Embodiment 2 is different from Embodiment 1 in that images to be 
combined are captured by different optical systems. When the images to be 
combined are captured by the same optical system, these images are 
subjected to shake correction and then combined, whereby a high-resolution 
image was obtained. 

[0122] 

However, when the images to be combined are captured by different 
optical systems, a parallax occurs because the optical systems are different. 
The present embodiment is to correct the parallax when the images to be 
combined are captured by different optical systems. Hereinafter, an 
explanation will be given with reference to examples. 

[0123] 

(Example 4) 

FIG. 11 illustrates a configuration of an imaging optical system, a 
pixel shift means, and an imaging element according to Example 4. As 
imaging optical systems, aspherical lenses 1101a to 110 Id each having a 
diameter of 3 mm are used. Each lens has an optical axis substantially 
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parallel to a Z axis in FIG. 11. Color filters 1102a to 1102d are provided 
before the respective lenses (on the subject side) as wavelength separating 
means that transmit only specific wavelengths. 1102a and 1102d indicate 
color filters which transmit a green color, 1102b indicates a color filter which 
transmits a red color, and 1102c indicates a color filter which transmits a 
blue color. 1103a to 1103d indicate four imaging elements corresponding to 
the respective lenses in one-to-one correspondence, and are configured to 
receive only specific color components. By combining images formed by the 
optical systems (color components), a color image can be produced. 
[0124] 

The imaging element has a pixel pitch of 3 \xm in this example. The 
lens and the imaging elements are each provided parallel to an X axis in 
FIG. 11 and are equally spaced, and each imaging element has a light 
receiving surface parallel to the XY plane in FIG. 11. 1104 indicates a 
piezoelectric small movement mechanism that serves as a pixel shift means. 

[0125] 

The imaging elements 1103a to 1103c are attached to the 
piezoelectric small movement mechanism so that the imaging elements 
1103a to 1103c are driven in the X direction and the Y direction in FIG. 11. 
1103d is separated from the piezoelectric small movement mechanism, and 
is configured not to perform pixel shift. 

[0126] 

FIG. 12 is a top view of the piezoelectric small movement 
mechanism. The imaging elements are provided on a stage 1201 at a 
center portion. The stage 1201 is moved finely in the X-axis direction of 
FIG. 12 using laminated piezoelectric elements 1202a and 1202b, and a 
stage fixing frame 1202 is moved finely in the Y-axis direction of FIG. 12 
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using laminated piezoelectric elements 1203a to 1203d. Thereby, the 
imaging element can be moved finely separately in two axial directions 
orthogonal to each other in a horizontal plane of the imaging element. 
[0127] 

In this example, by one image capture command, four images were 
captured while performing pixel shift. The imaging elements were 
configured to perform image capture while being shifted by a 1/2-pixel pitch 
(1.5 jam) in each of the X direction and the Y direction. Specifically, an 
image capture was performed without pixel shift (first image), and then the 
imaging elements were shifted by a 1/2 pitch in the X direction to perform 
an image capture (second image). Next, the imaging elements were shifted 
by a 1/2 pitch in the Y direction while keeping the X-direction position to 
perform an image capture (third image). Finally, the imaging elements 
were moved by a -1/2 pitch in the X direction while keeping the Y- direction 
position to perform an image capture (fourth image). By combining these 
four images, a high-resolution image was obtained. 

[0128] 

From a plurality of images captured in time series using the imaging 
optical system llOld that does not perform pixel shift, shake amounts at 
respective image capture times were obtained. Based on the obtained 
shake amounts, the optimal image selecting means 109 selected images 
captured by the optical system that performs pixel shift. A method 
illustrated in FIG. 4 was used as a method of selecting images. With 
respect to the images, each of which is captured by performing the pixel 
shift using the imaging optical systems 1101a, 1101b, and 1101c each 
corresponding to green, red, and blue colors, a shake correction was 
performed for each. In the imaging apparatus of present example, as will 
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be described later, after parallax amounts are corrected for the images 
captured at the same time by the different imaging optical systems 1101a to 
1101c, the images are combined, thereby producing a full-color image. 
[0129] 

In addition, if the image has a different shake amount, the image 
selected by the optimal image selecting means 109 may possibly be an 
image captured by the imaging optical system llOld that does not perform 
pixel shift. The present example does not include an imaging optical 
system that does not perform pixel shift and corresponds to red and blue 
colors. Accordingly, it is possible that a color is created for an optical 
system other than the existing optical system through calculation to be 
combined. 

[0130] 

Also, it is possible that an image of the optical system that performs 
pixel shift is selected regardless of the shake amount. In such a case, the 
effect of pixel shift may be reduced depending on a shake amount but it will 
not be significantly disadvantageous in correcting the shake amount. 

[0131] 

If an imaging optical system that does not perform pixel shift is 
provided for each of all green, red and blue colors, high-resolution image can 
be obtained reliably even if images selected by the optimal image selecting 
means 109 are of the imaging optical system that does not perform pixel 
shift. 

[0132] 

Next, when an image of a subject is captured by different optical 
systems, the relative position of subject images formed on the imaging 
element varies (parallax occurs) depending on a distance to the subject. 
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[0133] 

Hereinafter, a parallax will be explained with reference to FIG. 13. 
For the sake of simplicity, two imaging optical systems 1301a and 1301b 
having the same characteristics are provided at locations separated from 
each other by a distance D, and have image formation surfaces 1302a and 
1302b, respectively. In this case, the imaging optical systems 1301a and 
1301b observe the same subject from different positions. Therefore, a 
parallax occurs between images formed on the image formation surfaces 
1302a and 1302b. 

[0134] 

A parallax amount A is given by (expression 2). D indicates an 
interval between an optical axis of the imaging optical system 1301a and an 
optical axis of the imaging optical system 1301b, f indicates a focal length of 
the imaging optical systems 1301a and 1301b, and A indicates a distance 
between a subject and the imaging optical systems 1301a and 1301b. 

[0135] 

[Expression 2] 
A = Df/(A-f) 

[0136] 

When A is sufficiently large so that the subject is assumed to be 
located at infinity, the parallax amount A can be represented by D-&A, and A 
can be assumed to be 0. In this case, images captured by the imaging 
optical systems 1301a and 1301b can be assumed to be the same. 
Therefore, the images can be combined as they are. 

[0137] 

However, if A is small, the parallax amount A is a finite value and is 
not negligible. Specifically, the images captured by the imaging optical 
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system 1301a and the imaging optical system 1301b have a difference due to 
a parallax, depending on the distance to a subject, and cannot be assumed to 
be the same. Therefore, the images cannot be superposed and combined as 
they are. 

[0138] 

In order to correct the parallax, an image capture with pixel shift is 
performed four times. The images captured first time by the lenses 1101a 
and llOld are divided into square blocks having 24x24 pixels. The images 
are compared and shift amount (parallax amount) is obtained using 
(expression l) in the same manner as in obtaining shake amount as 
mentioned above. In this case, if the square blocks for an image captured 
by the lens 1101a is set to be a comparative reference region, then the 
square blocks for an image captured by the lens 110 Id is set to be an 
evaluation region. 

[0139] 

A method of dividing into blocks is not limited to this, and the 
number of pixels or the shape of each divided block may be changed. 
[0140] 

As is different from the obtaining of a shake, a direction in which a 
parallax occurs is limited to a linear direction connecting the origins of the 
imaging elements (intersections of the imaging elements and the optical 
axes of the respective corresponding optical systems), and therefore, when a 
parallax is detected, a combination of m and n in (expression l) may be 
limited, depending on the direction. 

[0141] 

A distance A, a distance to the subject, is obtained at the same time 
based on the obtained parallax amount. With respect to the images 
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captured by the rest of the lenses 1101b and 1101c, the correction can be 
carried out by calculating a parallax amount from a distance to the position 
of the imaging optical systems. In this example, the lens and the imaging 
elements are both equally spaced. 
[0142] 

Accordingly, in (expression 2), a value of D between the lenses 1101a 
and 1101b and a value of D between the lenses 1101a and 1101c correspond 
to 1/3 and 2/3 of a value of D between the lenses 1101a and llOld, 
respectively. Therefore, the parallax amount in each of the optical systems 
1101b and 1101c corresponds to 1/3 and 2/3 of the parallax amount between 
the lenses 1101a and llOld, respectively. 

[0143] 

A shake amount is corrected with respect to each of the images 
captured with pixel shift as mentioned above, and the images that 
correspond to the respective colors (the images captured in time series by 
the same optical system) are combined for each color, whereby a 
high-resolution single color image can be obtained for the three colors. 

[0144] 

Next, when high-resolution single color images corresponding to 
each color (the images captured by the different optical systems) are 
combined, parallaxes occurred between the high resolution single color 
images are corrected based on a parallax amount obtained by the parallax 
obtaining means as mentioned above. Then, by carrying out a combination 
process with respect to the high-resolution single color images for the three 
colors, a full-color image of high resolution can be obtained. 

[0145] 

The order of combining images is not limited to the above-mentioned 
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order. For instance, images of different color optical systems captured by 
performing pixel shift at the same time (for example, image capture time l) 
are combined after the parallax correction, producing a full-color image. 
Next, the produced full-color image is stored into a memory. Then, images 
of different color optical systems captured by performing pixel shift at a 
different time (for example, image capture time 2) are combined similarly, so 
that a full-color image is obtained to be stored into a memory. The 
full-color images stored into the memory on the pixel shift basis (on the 
variant time basis) are combined eventually, thereby a full-color image of 
high resolution is obtained. 
[0146] 

The parallax amount obtained in this way allows accurate correction 
of images when the images of different optical systems are combined, and 
these images corrected are combined respectively. This results in a color 
image of high resolution to be produced. 

[0147] 

In addition, it is not necessary to arrange the four optical systems to 
be aligned. FIG. 14A shows another example in arranging the four optical 
systems. In this example in FIG. 14A, the four optical systems are 
provided to form a rectangle. GO and Gl represent green, R represents red, 
and B represents blue. 

[0148] In this case, for obtaining of a parallax amount, 

parallaxes of the green imaging systems that are diagonally provided are 
used. Parallax amounts of the remaining red-color and blue-color optical 
systems are orthogonal components of the parallax amount of the green 
optical system previously obtained since the optical systems are provided to 
substantially form a rectangle as shown in the figure (see FIG. 14B). 
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[0149] Although, in this example, the color filter is provided 

before the lens to separate a wavelength, the color filter may be provided 
between the lens and the imaging element, or the color filter may be formed 
directly on the lens. 

[0150] 

The color filters are not necessarily limited to the three primary 
colors R, G and B. Complementary color filters may be used to separate 
wavelengths, and color information may be reversed and combined by image 
processing. 

[0151] 

The wavelength separating means is not limited to color filters. 
For example, when a glass plate is used as the pixel shift means and a 
mechanism of tilting the glass plate is used, a colored glass may be used as 
the glass plate. Thus, any specific means may be used as the wavelength 
separating means as long as it is a means for separating a predetermined 
wavelength component. 

[0152] 

Furthermore, the present example provides information on the 
distance to a subject by calculation of parallax. This makes it possible to 
discriminate a subject within an image as described in Example 3. In other 
words, a distance obtained from a parallax discriminating means can be 
used for a subject discriminating means. Accordingly, by obtaining a shake 
in an image for each of the discriminated subjects, a shake amount can be 
obtained more specifically. 

[0153] 

Although it has been described as an example that images captured 
by the optical system handling a green color are compared to obtain parallax 



42 



and shake amounts, the color is not necessarily limited to green and it is 
possible to obtain a similar effect in any color. 
[0154] 

Furthermore, as mentioned above, the optical system that does not 
perform pixel shift may be provided to correspond to any of the color 
components so as to improve a resolution. In such a case, obtaining of 
shake amount also can be reliable. For instance, in a case where there is a 
subject shake in a fully blue-colored substance, when a shake amount is 
obtained only by the optical system handling a green color, information on 
the blue colored substance is lost owing to a wavelength separating means 
and thus, a shake amount cannot be obtained. At the same time, the 
optical system that does not perform pixel shift is provided to correspond to 
any of the color components, so that a comparison between shake amounts 
for each color component is possible. This prevents the information loss on 
the shake amount and provides further reliable shake correction. 

[0155] 

Also, in this example, it may be possible that a filter 1102d is 
provided on an optical axis of the lens llOld when a parallax correction is 
performed and a filter 1102d is provided off the optical axis of the lens llOld 
when a pixel shift is performed. With such a configuration, two images of 
green color that are assumed to be completely identical at the parallax 
correction are to be compared. Therefore, parallax can be obtained with 
high precision. 

[0156] 

At the pixel shift, the removal of filter causes imaging elements that 
correspond to the respective optical systems to be equivalent to imaging 
elements of monochrome images. With this configuration, even there is a 
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subject shake in a fully blue-colored substance, for example, the shake is 
obtained as luminance information, whereby the accurate shake correction 
can be performed. 
[0157] 

(Embodiment 3) 

Embodiment 3 is different from Embodiments 1 and 2 in that a shift 
amount determining means for determining a shift amount for a pixel shift 
is used. Accordingly, a flowchart of image capture in the present 
embodiment is also different from those of Embodiments 1 and 2. 

[0158] 

According to a flowchart of FIG. 2 as described in Embodiment 1, 
when an image capture for a pixel shift is finished, the captured images are 
compared to obtain a shake amount. Then, optimal images for performing 
a combination of images are selected by an optimal image selecting means 
109. In this embodiment, a shift amount for pixel shift is determined by 
the shift amount determining means before a pixel shift is performed. 
Thus, a flowchart as shown in FIG. 15 is used rather than a flowchart in 
FIG. 2. 

[0159] 

Hereinafter, the present embodiment will be described in accordance 
with FIG. 15. A basic configuration of the imaging apparatus is similar to 
that in Embodiment 1 or Embodiment 2, and the shift amount determining 
means will not be described in a description using FIG. 1. 

[0160] 

Since steps 1500 and 1501 are similar to steps 200 and 201 of FIG. 2 
as described in Embodiment 1, they will not be described. Next, in step 
1502, an image capture is performed. Step 1502 includes steps 1503 to 
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1508 as one unit. 
[0161] 

Steps 1503 and 1504 are similar to steps 203 and 204 of FIG. 2 as 
described in Embodiment 1. In step 1505, images captured by a system 
that does not perform pixel shift and stored into a memory are compared to 
obtain a shake amount. As described in Embodiment 1, a shake amount 
may be obtained block by block or from an entire image. A shake amount 
may also be obtained for each subject. 

[0162] 

In step 1506, the optimal image selecting means selects the optimal 
image for combining of images based on the obtained shake amount. For a 
method for selecting the optimal image, a method described in Embodiment 
1 with reference to FIG. 4 can be used. In step 1507, shift amounts for 
performing subsequent pixel shift are determined based on the shake 
amount obtained in step 1505 and the image selected in step 1506. 

[0163] 

In step 1508, a pixel shift is performed by a pixel shift means 102 
based on the shift amount determined in step 1507. Hereinafter, a series of 
steps in step 1502 are performed repeatedly until a predetermined number 
of times is finished. When the image capture is finished, in step 1509, a 
parallax amount of the image selected by the optimal image selecting means 
is obtained using a parallax obtaining means. Then, images are corrected 
based on the shake amount and the parallax amount obtained in step 1510 
and are combined and then, an image is output in step 1511. This 
completes a series of image capture operation. Hereinafter, a detailed 
description will be provided with reference to an example. 

[0164] 
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(Example 5) 

FIG. 16 illustrates an imaging optical system, a pixel shift means, 
and an imaging element according to Embodiment 5. As compared with 
Example 1 of FIG. 6, in a basic configuration of the present example, lenses 
1601a, 1601b and an imaging element 1603 of an imaging optical system, 
similar to those of Example 1, are used. However, the present example is 
different in that two glass plates 1602a, 1602b are used as the pixel shift 
means. 

[0165] 

A method for pixel shift using glass plate is similar to the method 
indicated in Example 1. However, the present example makes it possible to 
perform pixel shift in the X direction and the Y direction in FIG. 16 by 
tilting the glass plates. A tilting means is not illustrated in FIG. 16. In 
this example, a pixel shift in the X direction is performed by tilting the glass 
plate 1602a and a pixel shift in the Y direction is performed by tilting the 
glass plate 1602b. For each of the glass plates 1602a and 1602b, BK7 
similar to that in Example 1 is used. 

[0166] 

In this example, images are captured four times by performing pixel 
shift. FIG. 17 shows a sequence of performing pixel shift. In the pixel 
shift, a shift amount is set to be a half pixel pitch both in the horizontal and 
vertical direction of the imaging element, which is 1.2 jim. In an example 
of FIG. 17A, an image is captured without pixel shift (first image) and then, 
an image is captured by tilting the glass plate 1602a in the X direction by an 
amount that corresponds to 1/2 pitch (second image). Next, an image is 
captured by tilting the glass plate 1602b in the Y direction by an amount 
that corresponds to a 1/2 pitch while keeping the position of the glass plate 
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1602a the same (third image). Finally, an image is captured by tilting the 
glass plate 1602a in the X direction by an amount that corresponds to a -1/2 
pitch while keeping the position of the glass plate 1602b the same (fourth 
image). By doing this, the imaged state in which invalid portions in each of 
the pixels are interpolated as in FIG. 17B can be obtained. 
[0167] 

In the present example, an image capture is performed for a scene in 
which movement of a subject is sufficiently small (for example, landscape 
and the like). Therefore, it is free from a subject shake and even when 
there is a shake, it may be assumed that the shake is resulted from a move 
of the entire image due to a photographer shake. A shake amount is 
obtained in a manner similar to that in Example 1. 

[0168] 

First, a result of image capture without using a shift amount 
determining means is shown in FIG. 18 using the flowchart of FIG. 2 in 
Embodiment 1. In FIG. 18, an intended shift amount of pixel shift (shift 
amount in FIG. 18 ), a shake amount obtained by each image capture 
(optical system without pixel shift in FIG. 18), and an image captured by an 
optical system with pixel shift (optical system with pixel shift in FIG. 18) 
are illustrated together. 

[0169] 

In FIG. 18, a shift amount of pixel shift is represented by a vector in 
solid lines and a shake amount obtained is represented by a vector in broken 
lines. Although each of the vectors appears only in an optical-to-electrical 
conversion portion at the upper left end, a shift due to a pixel shift and a 
shake during an image capture are supposed to be present equally for the 
same vector in all of the image capture regions. Hereinafter, a description 
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will be given in accordance with a flow in the image capture using FIG. 18. 
[0170] 

First, image capture was performed to obtain four images in the 
order of pixel shift as shown in FIG. 17. Next, an image captured by an 
optical system that does not perform pixel shift was compared to obtain a 
shake amount. The obtained shake amount is as shown in FIG. 18. 

[0171] 

No shake was obtained in the first image captured but in the second 
image captured, with respect to the optical system that does not perform 
pixel shift, there was a shake in the entire image at a position in FIG. 17 
where the third image capture was intended. In such a case, with respect 
to the optical system that performs pixel shift, an image capture was 
performed at the position in FIG. 17 where the fourth image capture was 
intended (see third column in a second image's row in FIG. 18). 

[0172] 

For the third image captured, there was a shake in the entire image 
at the optical-to-electrical conversion portion with respect to the optical 
system that does not perform pixel shift. In such a case, with respect to the 
optical system that performs pixel shift, an image capture was performed at 
the position in FIG. 17 where the third image capture was intended (see 
third column in a third image's row in FIG. 18). 

[0173] 

For the fourth image captured, there was a shake in the entire 
image at the optical-to-electrical conversion portion with respect to the 
optical system that does not perform pixel shift. In such a case, with 
respect to the optical system that performs pixel shift, an image capture was 
performed at the position in FIG. 17 where the fourth image capture was 
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intended (see third column in a fourth image's row in FIG. 18). 
[0174] 

Next, based on the obtained shake amount, an image that is optimal 
for combining was selected using an optimal image selecting means. 
Information on the image obtained by image capture are only the first, third, 
and fourth image capture intended in FIG. 17 as described above. Thus, 
the optimal image selecting means selected images with shaded area from 
among the images of the optical system that performed pixel shift in FIG. 18. 
The selected images were then corrected based on the shake amount 
obtained to combine images. The combined image was output in the end, 
which completes a series of image captures. 

[0175] 

By performing the pixel shift arid subsequent combining of images 
using the optimal image selecting means in this way, the image improves a 
resolution reliably. However, because the combining of images was 
performed while information on the second image capture intended in FIG. 
17 was missing, it failed to achieve a resolution as substantially expected. 

[0176] 

In this regard, an image capture was performed in the present 
example using a flowchart as shown in FIG. 15. The result of image 
capture is illustrated in FIG. 19. Hereinafter, a description will be given in 
accordance with a flow in the image capture using FIG. 19. 

[0177] 

First, the first image was captured. When an attempt was made to 
obtain a shake amount using the shake amount obtaining means 108, no 
shake was obtained for the first image capture. Based on the obtained 
shake amount, an image from the first image capture with a shade region in 
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FIG. 19 was selected by the optimal image selecting means 109. Next, a 
shift amount for the second image capture was determined using the shift 
amount determining means to be set at the position in FIG. 17 where the 
second image capture was performed as intended. 
[0178] 

Next, the second image was captured. When an attempt was made 
to obtain a shake amount using the shake amount obtaining means 108, a 
shake was obtained at the position in FIG. 17 where the third image capture 
was intended in the optical system that does not perform pixel shift in the 
second image captured. In such a case, with respect to the optical system 
that performs pixel shift, an image capture was performed at the position in 
FIG. 17 where the fourth image capture was intended (see the third column 
in a second image's row in FIG. 19). Based on the obtained shake amount, 
an image captured by the optical system that does not perform pixel shift in 
the second image captured and an image captured by the optical system 
that performs pixel shift that are illustrated in FIG. 19 with shaded region 
were both selected using the optimal image selecting means 109. 

[0179] 

Next, using a shift amount determining means, a shift amount for 
pixel shift in the third image was determined to be set at a position in FIG. 
17 for the second image capture where an image capture were not possible. 

[0180] 

Next, the third image was captured. When an attempt was made to 
obtain a shake amount using the shake amount obtaining means, a shake 
was obtained at the optical-to-electrical conversion portion with respect to 
the optical system that does not perform pixel shift. In such a case, with 
respect to the optical system that performs pixel shift, an image capture was 
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performed at the position in FIG. 17 where the second image capture was 
intended (see the third column in a third image's row in FIG. 19). Based on 
the obtained shake amount, an image that is illustrated in FIG. 19 with 
shaded region was selected using the optimal image selecting means. 
[0181] 

By the end of the third image capture, information on four images 
that are intended to be captured were provided, so that the image capture 
was terminated by the third image capture. Next, based on the shake 
amount and parallax amount obtained in each image capture, the image 
selected by the optimal image selecting means was corrected to perform 
combining of images by the image combining means, thereby an image with 
improved resolution was obtained. 

[0182] 

By using the shift amount determining means, the shift amount is 
varied each time the pixel shift is performed to capture an image even when 
the image capture is not performed as intended at the beginning, whereby 
an image with improved resolution is obtained. 

[0183] 

In addition, in this example, each of the image capture is performed 
followed by determining of a shift amount for a pixel shift using a shift 
amount determining means to perform the next image capture. However, 
it may also be possible to perform image capture for a plurality of images 
before determining a shift amount for a pixel shift in the subsequent image 
capture using a shift amount determining means. In such a case, shift 
amounts for the subsequent image capture can be determined using 
information on images captured in a series, which saves time for image 
capture. 
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[0184] 

Furthermore, a pixel shift in the X direction is performed using a 
glass plate 1602a and a pixel shift in the Y direction is performed using a 
glass plate 1602b in the present example. However, it is possible that the 
respective pixel shift may be performed in the other direction. 

[0185] 

Furthermore, in the present example, although a pixel shift in the X 
and Y directions are performed using two glass plates, the pixel shift may be 
performed such that one glass plate is used and is actuated separately in 
the X and Y directions using the tilting means. 

[0186] 

Although, in this example, a method of tilting the glass plate is used 
as the pixel shift means, the method is not limited thereto. For example, 
an actuator employing a piezoelectric element, an electromagnetic actuator, 
or the like may be used to mechanically move the imaging element or the 
lens in a predetermined amount. 

[0187] 

In this example, a glass plate is not provided in the optical system 
that does not perform pixel shift. However, a pixel shift is not performed 
by the optical system provided with the glass plate which is not intended to 
be tilted. With this configuration, more flexibility will be available in a 
choice between with and without the pixel shift compared to the 
configuration in which the pixel shift means is not provided from the first 
place. 

[0188] 

This configuration is the same for another method of performing 
pixel shift as mentioned above. For example, when an actuator employing 
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a piezoelectric element is used as a pixel shift means, a pixel shift is 
intended not to be performed by not providing an actuator from the first 
place. However, it is also possible that the pixel shift is intended not to be 
performed by providing an actuator which is not intentionally moved. As 
in the case of the above-mentioned parallel plane, this case provides more 
flexibility in a choice between with and without the pixel shift compared to 
the configuration in which the pixel shift means is not provided from the 
first place. 

[0189] 

Although, in this example, one imaging element is divided into two 
regions, two different imaging elements may be employed for the respective 
optical systems in one-to-one correspondence. Any form of imaging 
element may be used as long as a plurality of imaging regions corresponds 
to respective optical systems in one-to-one correspondence. 

[0190] 

In the present example, an image capture is performed in a situation 
where it is only a photographer shake that can be assumed of a shake, and 
where there is a shake evenly in the entire image. However, the present 
example also includes a case where there is a subject shake and the shake 
amount of the entire image is not even. For example, when an image 
selected by the optimal image selecting means 109 is different for different 
block or different subject, a shift amount determined by the shift amount 
selecting means can be selected so that some of the images are optimized. 

[0191] 

For example, a shift amount can be selected so as to make a specific 
subject (human) have a high resolution. In such a case, it is possible that a 
portion of the image excluding some of the images being selected is not 
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improved of a resolution by pixel shift, and only the first image captured is 
selected. 

[0192] 

FIG. 20 shows a result of image capture according to another 
example where an image is captured using a flowchart illustrated in FIG. 15. 
An example shown in FIG. 20 is a case where four times of image capture 
was performed in an order illustrated in FIG. 17 and an attempt was made 
to improve a resolution. There was no shake obtained until the third image 
capture and the image capture was performed as intended, but there was a 
shake in the fourth image capture as shown in FIG. 20. 

[0193] 

In this case, the only information obtained up to the fourth image 
capture was on the first, second, and third image captures as intended. 
Accordingly, combining of these images is performed while the fourth image 
intended to be captured is missing, so that the improvement of a resolution 
is not achieved as intended. 

[0194] 

Consequently, in an example as shown in FIG. 20, image capture 
was performed until the fourth image intended to be captured in FIG. 17 
can be obtained. A shake was obtained as shown in FIG. 20 when the fifth 
image was captured. On the fifth image capture, an image of an optical 
system without pixel shift corresponds to the fourth image intended to be 
captured in FIG. 17. Thus, the optimal image selecting means 109 selected 
an image captured without pixel shift from the fifth image capture, thereby 
completing the image capture. 

[0195] 

Accordingly, when an intended image cannot be obtained in 
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improving of a resolution using pixel shift technique, the image capture may 
be performed repeatedly until the intended image is obtained. Also, this 
can be applied not only to a case described above with reference to FIG. 20 
where the last image capture alone is an unintended image but also a case 
where the image captures along the way was not performed as intended. 
[Industrial Applicability] 
[0196] 

As described above, according to the present invention, even if there 
is a photographer shake or a subject shake when pixel shift is performed, it 
is possible to reduce a reduction in effect of pixel shift, and obtain a 
high-resolution image. Therefore, the present invention is useful for, for 
example, an imaging apparatus used in a digital still camera, a mobile 
telephone, or the like. 
[Brief Description of the Drawings] 

[0197] 

[Fig. l] FIG. 1 is a block diagram illustrating a configuration of an 
imaging apparatus according to Embodiment 1 of the present invention. 

[Fig. 2] FIG. 2 is a flowchart illustrating a whole operation of the 
imaging apparatus of Embodiment 1 of the present invention. 

[Fig. 3] FIG. 3 is a diagram illustrating a positional relationship 
between a comparative reference region and an evaluation region according 
to an embodiment of the present invention. 

[Fig. 4] FIG. 4 is a conceptual diagram of a method of selecting an 
optimal image according to an embodiment of the present invention. 

[Fig. 5] FIG. 5 is a conceptual diagram of a method of correcting a 
shake amount according to an embodiment of the present invention. 

[Fig. 6] FIG. 6 is a diagram illustrating a configuration of an 
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imaging optical system, a pixel shift means, and an imaging element 
according to Example 1 of the present invention. 

[Fig. 7] FIG. 7 is a diagram illustrating an image which is stored 
into a memory in Example 1 of the present invention. 

[Fig. 8] FIG 8 is a conceptual diagram showing an example of a 
photographer shake. 

[Fig. 9] FIG. 9 is a diagram illustrating an image which is stored 
into an memory in Example 2 of the present invention. 

[Fig. 10] FIG. 10 is a diagram illustrating a captured image and 
subject groups discriminated by a subject discriminating means in Example 
3 of the present invention. 

[Fig. 11] FIG. 11 is a diagram illustrating a configuration of an 
imaging optical system, a pixel shift means, and an imaging element 
according to Embodiment 2 of the present invention. 

[Fig. 12] FIG. 12 is a top view of a piezoelectric small movement 
mechanism according to Example 4 of the present invention. 

[Fig. 13] FIG. 13 is a diagram for explaining a parallax. 

[Fig. 14] FIG. 14 is a diagram illustrating an exemplary 
arrangement of optical systems according to an embodiment of the present 
invention. 

[Fig. 15] FIG. 15 is a flowchart illustrating a whole operation of an 
imaging apparatus of Embodiment 3 of the present invention. 

[Fig. 16] FIG. 16 is a diagram illustrating a configuration of an 
imaging optical system, a pixel shift means, and an imaging element 
according to Example 5 of the present invention. 

[Fig. 17] FIG. 17 is a diagram illustrating a sequence of image 
capture according to Example 5 of the present invention. 
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[Fig. 18] FIG. 18 is a diagram illustrating a result of image capture 
performed using a flowchart of FIG. 2 in Embodiment 5 of the present 
invention. 

[Fig. 19] FIG. 19 is a diagram illustrating a result of image capture 
performed using a flowchart of FIG. 11 in Embodiment 5 of the present 
invention. 

[Fig. 20] FIG. 20 is a diagram illustrating a result of image capture 
performed using a flowchart of FIG. 11 in Embodiment 5 of the present 
invention. 

[Fig. 21] FIG. 21 is a conceptual diagram illustrating an example of 
improving a resolution using a conventional pixel shift technique. 

[Fig. 22] FIG. 22 is a diagram for explaining a problem in the 
conventional pixel shift technique. 

[Fig. 23] FIG. 23 is a diagram for explaining a problem in the 
conventional pixel shift technique. 
[Description of the Reference Numerals] 

[0198] 
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[Document Name] ABSTRACT 
[Abstract] 

[Objective] To provide an imaging apparatus that performs a pixel shift and 
can prevent a reduction in the effect of the pixel shift even if there is a 
photographer shake or a subject shake. 

[Means for Solving the Problem] A pixel shift means 102 does not perform a 
pixel shift on images formed by one of the optical systems 104, 105, but 
performs a pixel shift on images formed by the other optical system. A 
shake amount obtaining means 108 obtains the shake amount of an image 
by comparing a plurality of pieces of image information captured in time 
series by the optical system without performing the pixel shift and stored in 
a memory 107. An optimal image selecting means 109 selects images used 
for image combination based on the shake amount obtained by the shake 
amount obtaining means 108. An image combining means 110 corrects the 
selected images based on the shake amount obtained by the shake amount 
obtaining means 108, and then combines the images. 
[Selected Figure] FIG. 1 
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[FIG. 2] 
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[FIG. 4] 




2005 154447 



4/20 



[FIG. 5] 



[FIG. 6] 



Image capture by imaging 
system 104 



Image capture by imaging 
system 105 



CO 



CD 



501a 



CO 

E 




502a 




502 



504f 



CD 
k. 
3 
■+-> 

Q.CM 



CD 



CO 

o 
co 




503 



503c 




504c 5Q4e 



504 



603a- 



/ v 
























* 
/ 














/ 

/ 


t 








/ 

/ 

✓ 






V 




\/^r\^601b 




602 



-603 




2005-154447 



5/20 



[FIG. 7] 
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[FIG. 8] 
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[FIG. 9] 
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[FIG. 10] 
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[FIG. 12] 
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[FIG. 15] 
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[FIG. 16] 
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[FIG. 17] 
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[FIG. 18] 
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[FIG. 19] 
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[FIG. 20] 
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[FIG. 23] 




